DNFStore: A Distributed Netflow Storage System Supports Fast Retrieval

Network anomaly detection or network optimization based on Netflow plays an important role in current high-speed network management. Storage and analysis of high-speed continuous Netflow are hot and difficult issues in network security research and industry communities. Existing solutions, although useful in above areas, have several drawbacks in well handling Netflow records generated by large-scale backbone networks. In this paper, we design and implement distributed Netflow storage system DNFStore. First, DNFStore dispatches high-speed Netflow records to efficient storage nodes through independent dispatchers to achieve parallel storage and uses interactive protocol to realize the goal of dynamically join and exit of nodes. Second, DNFStore merges Netflow records returned from storage nodes through query agent, and provides users with a fast unified query interface. We deploy DNFStore on a Gigabit network. The experimental results show that DNFStore can handle 20 million Netflow records per second and process multiple backbone networks concurrently. Besides, DNFStore can provide 40 times speed of query response than centralized storage under the same conditions.